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PREFACE 


This  report  addresses  completed  and  continuing  Arroyo  Center  work 
on  problems  in  ammunition  distribution  management  that  are  related 
to  information  management  and  decisionmaking  within  a  wartime 
theater  of  operations.  The  objective  of  the  project,  “Decision  Support 
Systems  for  Combat  Service  Support,’’  is  to  improve  combat  service 
support  (CSS)  performance  and  thereby  to  increase  the  combat  capa¬ 
bility  of  supported  forces  by  identifying  and  evaluating  applications  of 
decision  support  technology  that  could  improve  the  quality  or  timeli¬ 
ness  of  CSS  management  decisionmaking.  The  research  falls  within 
the  Arroyo  Center’s  Readiness  and  Sustainability  Program,  and  is  part 
of  a  larger  effort  to  improve  the  Army’s  warfighting  capabilities  by- 
increasing  the  effectiveness  of  logistics  resource  management.  The 
project  is  sponsored  by  the  Commanding  General,  U.S.  Army  Logistics 
Center,  Fort  Lee,  Virginia. 

The  report  provides  an  overview  of  the  initial  stages  of  the  research 
and  describes  research  directions  and  objectives.  It  should  be  of 
interest  to  organizations  and  individuals  concerned  with  developin;, 
doctrine  for  ammunition  distribution  or  with  designing  and  evaluating 
information  management  and  decision  support  software.  Individuals 
involved  with  operations  research — especially  simulation  modeling, 
expert  systems,  and  decision  support  systems — may  also  be  interested 
in  the  analysis  techniques  described  here. 


THE  ARROYO  CENTER 

The  Arroyo  Center  is  the  U.S.  Army’s  federally  funded  research  and 
development  center  for  studies  and  analysis  operated  by  The  RAND 
Corporation.  The  Arroyo  Center  provides  the  Army  with  objective, 
independent  analytic  research  on  major  policy  and  management  con¬ 
cerns,  emphasizing  mid-  to  long-range  problems.  Its  research  is  carried 
out  in  five  programs:  Policy  and  Strategy;  Force  Deployment  and 
Employment;  Readiness  and  Sustainability;  Manpower,  Training,  and 
Performance;  and  Applied  Technology. 

Army  Regulation  5-21  contains  basic  policy  for  the  conduct  of  the 
Arroyo  Center.  The  Army  provides  continuing  gijidance  and  oversight 
through  the  Arroyo  Center  Policy  Committee,  which  is  co-chaired  by 
the  Vice  Chief  of  Staff  and  by  the  Assistant  Secretary  for  Research, 
Development,  ".r.d  Acquisition.  Arroyo  Center  work  is  performed  under 
contract  MDA903-86-C-0059. 
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The  Arroyo  Center  is  honsed  in  RAND’s  Army  Research  Division. 
The  RAND  Corporation  is  a  private,  non-profit  institution  that  con¬ 
ducts  analytic  research  on  a  wide  range  of  public  policy  matters  affect¬ 
ing  rhe  nation’s  security  and  welfare. 

Stephen  M.  Drezner  is  Vice  President  for  the  A* my  Research  Divi¬ 
sion  and  Director  of  the  Arroyo  Center.  Those  interested  in  further 
information  concerning  the  Arroyo  Center  should  contact  his  office 
directly: 


Stephen  M.  Drezner 
The  RAND  Corporation 
1700  Main  Street 
P.O.  Box  2138 

Santa  Monica,  California  90406-2138 
Telephone:  (213)  393-0411 


SUMMARY 


Future  wars  will  be  characterized  by  highly  dynamic  battlefields 
complicated  by  much  uncertainty  concerning  enemy  intentions  and 
actions,  required  operations  and  missions  to  counter  enemy  actions, 
and  weapon  system  reliability,  effectiveness,  and  survivability.  Logisti¬ 
cians  must  face  t^^e  uncertainties  of  war  and  the  variabilities  of 
demands  when  addressing  questions  concerning  the  quantity  of  various 
logistic  assets  required  at  different  locations  and  the  capabilities 
needed  to  move,  store,  and  maintain  those  assets.  Constrained  defense 
budgets  exacerbate  the  problems  faced  by  logisticians  and  eliminate  the 
alternative  of  “buying  out”  part  of  the  uncertainty  and  variability. 

The  uncertainty  and  variability  of  wartime,  coupled  with  constraints 
on  the  availability  of  logistic  assets,  will  require  flexible  and  responsive 
management  systems  that  can  allocate  the  limited  logistic  resources  in 
ways  that  maximize  combat  capability.  This  report  describes  the 
research  for  aii  Arroyo  Center  project  entitled  “Decision  Support  Sys- 
ten;s  for  Combat  Service  Support.”  The  objective  of  the  project  is  to 
investiga<’e  ways  of  increasing  combat  capability  by  improving  the 
effectiveness  of  logistic  resource  management.  Focusing  on  the  war¬ 
time  theater  ammunition  distribution  system  (WTADS),  we  are 
developing  methods  to  identify  and  evaluate  decision  support  enhance¬ 
ments  that  can  improve  the  quality  and  timeliness  of  management 
decisions. 


PHASE  I  RESEARCH  RESULTS 

The  initial  phase  of  the  research  identified  uncertainty  and  complex¬ 
ity  as  the  key  problems  facing  the  management  of  the  WTADS.  The 
general  solutions  to  these  problems  include: 

•  Better  information  and  short  time  horizon  forecasts 

•  Better  decision  support 

•  Increased  system  responsiveness. 

The  various  technologies  of  operations  research,  artificial  intelligence, 
computer  science,  and  decision  support  systems  were  examined  to  iden¬ 
tify  and  evaluate  potential  applications  that  could  help  provide  solutions 
to  the  overall  management  problems.  Three  high  priority  research  areas 
were  identified  during  the  first  phase  of  this  research.  First,  a  system  data 
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model  should  be  developed  to  understand  the  information  flows  and 
availabilities  within  the  WTADS.  Second,  a  quantitative  evaluation 
mechanism  was  required  to  measure  the  impact  of  potential  decision  sup¬ 
port  enhancements.  Third,  the  development  of  narrow-purpose  expert 
systems  offered  the  potential  for  improving  decision  support  within  the 
WTADS  and  warranted  further  investigation. 


PHASE  II  RESEARCH  RESULTS 

The  second  phase  of  the  research  evolved  into  two  parallel,  but 
related  paths  to  pursue  the  recommendations  made  during  the  first 
research  phase.' 

The  objective  of  the  first  path  was  the  development  of  a  prototype 
model  to  help  evaluate  the  combat  effects  of  proposed  decision  support 
tools.  The  resulting  knowledge-based  simulation  model,  written  in  an 
object-oriented  language,  represents  not  only  the  physical  distribution 
aspects  of  the  WTADS,  but  also  the  information  Hows  and  decision¬ 
making  capabilities  of  the  WTADS  managers.  The  simulation  expli¬ 
citly  models  materiel  management  centers,  movement  control  centers, 
and  various  other  ammunition  managers.  Furthermore,  tiie  WT.'\DS 
simulation  models  the  complete  ammunition  system  from  CONUS 
manufacturing  plants  to  the  ultimate  combat  users  (although  the  model 
concentrates  on  the  theater  distribution  of  ammunition). 

The  WTADS  model  captures  the  variability  and  uncertainty  of  war¬ 
time  by  allowing  planned  postures  and  target  sets  to  be  altered  on  a 
random  basis  and  by  allowing  the  enemy  to  interdict  and  destroy  logis¬ 
tics  resources.  Finally,  the  model  provides  not  only  standard  logistics 
measures  of  effectiveness,  but  also  combat-oriented  measures  so  that 
the  overall  effects  on  system  performance  can  be  measured  along  with 
the  local  effects  of  decisionmaking  enhancemt  r 

The  objective  of  the  second  path  of  the  research  was  the  develop¬ 
ment  of  a  method  to  identify  decisionmaking  problems  appropriate  for 
expert  system  solutions.  A  number  of  aspects  of  the  WTADS  suggest 
that  expert  systems  should  prove  beneficial  to  the  decisionmakers — 
several  decisions  involve  the  consideration  of  a  large  number  of  factors 
and  alternatives,  decisions  are  made  in  limited  time  frames,  the 
dynamic  and  stressful  environment  contribute  to  human  error,  and 
there  are  a  limited  number  of  trained  and  experienced  WTADS 


'Formal  development  and  specification  of  a  system  data  model  were  not  undertaken 
ba.sed  on  the  assumption  that  the  Army,  or  another  contractor,  could  adequately  accom 
plish  that  objective.  Understanding  and  representing  information  flows  were  critical 
aspects,  however,  of  developing  the  quantitative  evaluation  model. 
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managers.  The  problem  lies  in  identifying  those  applications  where 
expert  system  technolog>'  would  have  large  payoffs. 

The  resulting  expert  system  screening  mechanism  involves  a  wide 
range  of  questions  concerning  whether  a  problem  is  appropriate, 
whether  the  development  of  an  expert  system  is  feasible,  and  whether 
expert  system  development  efforts  can  be  justified.  Through  the 
interactions  of  a  knowledge  engineer  and  an  ammunition  system 
domain  expert,  a  number  of  WTADS  problems  were  identified  and  sub¬ 
jected  to  the  screening  criteria.  The  result  was  the  identification  of 
several  problems  that  were  applicable  to  expert  system  solutions,  but  a 
major  hurdle  to  the  development  of  expert  systems  was  the  lack  of 
domain  experts  with  relevant  wartime  experience. 


PHASE  III  RESEARCH  OBJECTIVES 

The  current  phase  of  our  research  is  pursuing  the  two  paths  ini¬ 
tiated  under  Phase  II  plus  examining  some  new  research  objectives. 
Now  that  the  prototype  WTADS  model  has  been  developed,  we  are 
using  it  to  examine  the  combat  effects  of  a  number  of  policy  and  struc¬ 
tural  changes  to  the  WTADS.  Specific  changes  being  evaluated 
include  the: 

•  Introduction  of  the  palletized  load  system  iPLS) 

•  Implementation  of  aspects  of  the  manuever-oriented  ammuni¬ 
tion  distribution  system  (MOADS) 

•  Examination  of  increased  management  of  high-lethality  muni¬ 
tions. 

The  expert  system  research  is  continuing  by  developing  a  prototype 
expert  system  for  one  of  the  approprit'e  problems  identified  by  the 
screening  mechanism.  We  are  also  addressing  the  shortage  of  relevant 
expertise  by  examining  methods  to  cultivate  new  knowledge  and  by 
investigating  the  potential  of  adaptive  expert  systems  that  learn  from 
actual  experience. 

The  new  research  questions  we  are  addressing  during  Phase  III 
include: 

•  Predevelopment  evaluation  of  decision  tools  in  order  to  quantify 
the  value  of  a  decision  aid  before  development  efforts  begin. 
The  ability  to  evaluate  at  this  stage  of  the  development  cycle 
permits  the  relative  ranking  of  several  potential  decision  aids 
and  can  be  used  to  help  justify  funding  for  development  efforts. 
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•  Postdevelopment  evaluation  of  decision  tools  in  order  to  under¬ 
stand  wh?ther  a  decision  aid  actually  improves  system  perfor¬ 
mance  and  whether  further  development  efforts  or  fielding  is 
justified. 

•  Implementation  of  decision  aids  in  order  to  determine  the  best 
strategy  for  interfacing  the  human  decisionmaker  with  the  deci¬ 
sion  tool.  Implementation  alternatives  would  include  what  data 
are  provided  by  the  decision  aid,  how  that  data  are  displayed  to 
the  decisionmaker,  and  what  mechanisms  are  available  for 
manipulating  data. 

A  numlier  of  the  research  issues  we  are  addressing  during  the 
current  phase  of  our  research  require  interfacing  a  human  decision¬ 
maker  with  the  WTAD£  simulation  model.  The  structure  of  the 
object-oriented  language  used  to  develop  the  WTADS  simulation  model 
provides  a  ready  and  effective  mechanism  for  such  human  interfaces. 
Any  of  the  objects  programmed  in  the  model  can  be  replaced  by 
interactive  inputs  in  terms  of  responses  to  messages  directed  to  that 
object.  Therefore,  a  human  can  be  seated  at  a  terminal  receiving 
model  output  and  provide  corresponding  input  representing  an  object's 
behaviors. 

To  provide  this  human/model  interface,  we  plan  to  build  a  labora¬ 
tory  environment  that  can  graphically  present  various  logistics  data 
and  battlefield  information  to  the  decisionmaker,  allow  him  to  make 
decisions,  and,  based  on  the  results  of  the  simulation  model,  see  the 
combat  effects  of  his  decisions.  This  type  of  capability  will  help  us  to 
cultivate  new  expertise,  examine  the  effects  of  various  implementation 
strategies,  and  make  postdevelopment  evaluations  of  decision  support 
tools.  Furthermore,  this  initial  laboratory  capability  can  be  extended 
to  include  other  functional  areas  in  addition  to  ammunition,  provide  an 
effective  training  device  for  WTADS  managers,  and  be  adapted  for  use 
as  a  logistics  wargaming  capability. 

It  is  important  to  note  that  our  research  does  not  have  a  narrow 
goal  of  developing  a  single  expert  system  for  WTADS  or  of  building  a 
laboratory  for  limited  use  in  this  research  environment.  Rather,  we  are 
striving  to  evolve  a  general  approach  for  developing  expert  systems  in 
domains  where  knowledge  is  scarce,  and  we  are  building  a  laboratc,y 
environment  that  is  portable  and  extensible  by  the  Army  for  training 
and  evaluation  purposes. 

Given  shrinking  defense  budgets,  improved  information  and  deci¬ 
sionmaking  capabilities  are  an  important  way  to  achieve  greater 
responsiveness  and  support  from  finite  logistic  resources.  The  develop¬ 
ment  of  decision  support  tools  and  enhancements  has  been  difficult  to 


fund  because  of  limited  research  and  development  budgets  and  prob 
lerris  with  quantifying  the  potential  benefits  of  the  resulting  manage¬ 
ment  tools.  The  techniques  and  models  developed  in  this  project 
should  help  the  Army  quantitatively  evaluate  the  impact  of  proposed 
decision  support  systems  and  thus  more  objectively  justify  iheir 
development. 
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I.  INTRODUCTION 


One  of  the  few  things  certain  about  war  is  that  unexpected  things 
happen.  The  deliberate  planning  process  in  peacetime  cannot  fully 
anticipate  enemy  intentions  or  actions,  the  specific  operations  and  mis¬ 
sions  that  will  be  required  to  counter  enemy  actions,  or  the  degree  of 
sunport  that  will  be  provided  by  allies.  This  uncertainty  in  wartime 
operations,  coupled  with  the  uncertainties  in  weapon  system  effective¬ 
ness,  reliability,  and  survivability,  result  in  difficult  problems  for  logis¬ 
tic  support  planners.  Logisticians  must  confront  these  uncertainties 
when  faced  with  decisions  concerning  not  only  the  required  quantities 
of  various  assets  such  as  repair,  repair  parts,  and  munitions,  but  also 
how  to  provide  the  storage  and  distribution  of  those  assets  to  ensure 
that  operational  plans  can  be  supported. 

Requirements  decisions  (how  much  is  needed)  are  addressed  by 
logisticians  in  peacetime  using  available  data  and  knowledge,  supple¬ 
mented  by  assumptions  concerning  wartime  operations  and  weapon 
system  characteristics.  Regardless  of  whether  these  decisions  are 
correct  or  incorrect,  the  result  is  a  fixed  set  of  logistic  assets  available 
at  the  outbreak  of  war.  Because  of  the  uncertainties  about  how  much 
will  actually  be  required  and  where  and  when  it  will  be  needed,  it  is 
particularly  important  that  logisticians  effectively  allocate  the  limited 
resources  in  conjunction  with  current  operational  plans  and  priorities. 
To  do  this,  flexible  and  responsive  management  and  distribution  sys¬ 
tems  are  necessary.  If  distribution  systems  can  be  developed  that  are 
more  responsive,  i.e.,  that  are  able  to  recognize  and  adapt  quickly  to 
unanticipated  events,  then  more  responsive  supply  of  key  commodities, 
such  as  spare  parts  and  munitions,  or  functions,  such  as  transportation 
or  repair,  may  overcome  the  problems  caused  by  the  wrong  mix  or 
quantities  of  those  commodities. 

Based  on  the  premise  that  wartime  variabilities  and  uncertainties 
will  require  flexible  and  responsive  management  systems,  the  Arroyo 
Center’s  Readiness  and  Sustainability  Program  is  conducting  a  project 
entitled  Decision  Support  Systems  for  Combat  Service  Support.  The 
objective  of  the  research  is  to  increase  combat  capability  by  improving 
the  effectiveness  of  logistics  resource  management.  Specifically,  we  are 
identifying  and  evaluating  decision  support  tools  and  technologies  that 
can  improve  the  quality  and  timeliness  of  management  decisions.  The 
initial  focus  for  the  study  is  the  wartime  theater  ammunition  distribu¬ 
tion  system  (WTADS). 
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PHASE  I  RESEARCH 

The  first  phase  of  the  study  identified  key  problems  related  to  infor¬ 
mation  management  and  decisionmaking  within  the  WTADS,  proposed 
general  solutions  to  those  problems,  and  considered  the  potential  con¬ 
tributions  the  technologies  of  artificial  intelligence  (AI),  computer  sci¬ 
ence  (CS),  and  operations  research  (OR)  could  make  in  supporting  gen¬ 
eral  solutions. 

The  two  key  problems  faced  by  the  managers  of  the  wartime  theater 
ammunition  distribution  system  were  identified  as  uncertainty  and 
complexity.  Uncertainty,  as  described  above,  arises  from  numerous 
sources — uncertainty  about  enemy  actions,  about  our  own  plans  and 
objectives,  about  demands  for  ammunition  and  the  supplies  of  not  only 
ammunition,  but  also  the  trucks,  material  handling  equipment,  and 
personnel  required  to  move,  store,  and  handle  ammunition,  and  uncer¬ 
tainty  about  uncontrollable  random  factors  such  as  the  weather.  The 
problems  associated  with  complexity  are  due  to  the  sheer  size  of  the 
WTADS  plus  the  amount  of  inter-player  and  inter-temporal  coordina¬ 
tion  required  for  many  actions  within  the  system.’ 

Three  general  solutions  were  identified  to  help  WTADS  managers 
address  these  key  management  problems: 

•  Better  information  and  short-term  forecasts 

•  Better  decision  support 

•  Increased  system  responsiveness. 

Better  information  and  short-term  forecasts  can  help  to  reduce  the 
uncertainty  inherent  in  wartime  operations.  Increased  decision  support 
capabilities  can  assist  WTADS  managers  in  dealing  with  the  complexi¬ 
ties  of  the  overall  distribution  system.  Finally,  increasing  the  respon¬ 
siveness  of  the  WTADS  will  allow  decisionmakers  to  recognize  prob¬ 
lems  and  identify  inconsistencies  earlier  and  will  assist  in  selecting  and 
implementing  solutions  to  those  problems  more  quickly. 

Within  these  three  general  solution  areas,  a  number  of  specific 
applications  of  AI,  OR,  and  CS  technologies  were  identified  as  high  lev¬ 
erage  research  objectives.  Two  of  these  research  objectives,  the 
development  of  a  prototype  WTADS  model  and  evaluation  mechanism 
and  the  development  of  an  enumeration  and  screening  mechanism  for 
expert  system  applications,  formed  the  basis  for  the  second  phase  of 
the  project. 

’Using  standard  Army  firing  rates,  the  ammunition  distribution  system  during  a 
major  war  will  move  across  the  theater  almost  an  aircraft  carrier's  tonnage  in  munitions 
every  day.  Also,  even  relatively  simple  operations  such  as  making  a  single  shipment  of 
ammunition  between  two  locations  require  38  separate  steps  to  be  taken  in  sequence  by 
seven  different  individuals  or  organizations  (see  Ref.  1,  p.  88). 
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OBJECTIVE  OF  THE  DOCUMENT 

This  report  reviews  the  research  accomplishments  of  the  second 
phase  of  the  project  and  describes  the  progress  and  objectives  of  the 
current  research  phase.  During  Phase  II  of  the  research,  we  developed 
a  prototype  model  for  use  in  evaluating  enhancements  to  the  ammuni¬ 
tion  distribution  system.  This  knowledge -based  simulation  model, 
written  in  an  object-oriented  language,  captures  not  only  the  physical 
distribution  aspects  of  the  WTADS,  but  also  the  information  flows  and 
management  decisionmaking  aspects.  The  model  incorporates  the 
uncertainty  of  wartime  and  provides  combat-oriented  measures  in  addi¬ 
tion  to  logistics-oriented  measures  of  effectiveness. 

A  second  output  of  the  Phase  II  research  is  an  expert  system  screen¬ 
ing  mechanism  for  identifying  decision  problems  that  are  appropriate 
for  expert  system  solutions.  Various  screening  criteria  were  applied  to 
a  range  of  WTADS  decision  problems,  with  several  identified  as  war¬ 
ranting  the  development  of  expert  systems.  A  major  problem,  however, 
is  the  apparent  lack  of  experts  with  relevant  wartime  experience. 

In  Phase  III,  we  are  extending  the  two  paths  of  our  current  research 
by  using  the  WTADS  model  to  evaluate  several  specific  structural 
changes  to  the  ammunition  distribution  system  and  by  investigating 
methods  to  develop  and  cultivate  management  expertise.  The  exis¬ 
tence  of  an  evaluation  mechanism  and  the  potential  for  the  develop¬ 
ment  and  application  of  expert  decision  aids  presents  new  research 
questions  and  opportunities.  These  include  the  post-development 
evaluation  of  specific  decision  tools  (does  a  decision  tool  really  improve 
combat  performance?),  the  identification  of  ways  to  interface  expert 
systems  with  WTADS  decisionmakers  (how  can  expert  systems  be 
implemented  to  optimize  their  use  by  and  benefit  to  decisionmakers?), 
and  the  predevelopment  evaluation  of  potential  decision  aids  (how  can 
the  potential  impact  on  combat  performance  of  a  decision  tool  be 
estimated  before  the  tool  is  ever  developed?). 


ORGANIZATION  OF  THE  DOCUMENT 

Section  II  reviews  the  development  of  the  WTADS  model— a 
knowledge-based  simulation  model  of  both  the  physical  distribution 
system — and  the  management  command  and  control  aspects  of  the 
ammunition  distribution  system,  and  describes  how  this  model  is  being 
used  to  evaluate  three  .specific  structural  changes  currently  being  con¬ 
sidered  by  the  Army.  Section  III  provides  a  similar  description  of  the 
expert  system  research,  reviewing  the  development  and  application  of  a 
problem  enumeration  and  screening  mechanism.  This  section  also 
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outlines  potential  methods  for  addressing  the  lack  of  meaningful  war¬ 
time  expertise  that  we  are  pursuing  in  the  current  research  phase  of 
the  studv.  Section  IV  describes  the  new  research  objectives  we  are 
undertak'ng  and  summarizes  the  concepts  and  operations  of  a  logistics 
laboratory — an  environment  to  interface  human  decisionmakers  with 
the  WTADS  model  that  is  essential  to  our  research. 


II.  DEVELOPMENT  OF  A  WTADS  MODEL 


The  size  and  complexity  of  the  WTADS  suggest  that  an  overall  sys¬ 
tem  model  is  required  to  evaluate  the  effects  of  alternative  strategies, 
policies,  and  structural  changes,  and  the  effects  of  better  information 
and  decision  support  tools.  This  section  describes  the  development  of 
a  prototype  system  model.  We  discuss  the  rationale  for  developing  a 
new  model  of  the  ammunition  distribution  system,  the  philosophy  we 
adopted  in  developing  the  model,  and,  in  a  brief  overview,  the  composi¬ 
tion  of  the  WTADS  model.  We  conclude  the  section  by  giving  an 
example  of  how  the  WTADS  model  can  be  used  to  evaluate  the  effects 
of  improved  decision  support  and  discuss  several  specific  possible  struc¬ 
tural  changes  to  the  ammunition  distribution  system. 


RATIONALE  FOR  DEVELOPING  THE  WTADS  MODEL 

The  first  phase  of  our  research  identified  numerous  decision  suppon 
tools  that  could  potentially  improve  the  management  of  the  ammunitio' 
distribution  system.  The  Army  is  also  considering  several  structural  and 
doctrinal  changes  to  the  overall  distribution  system.  Identifying  changes 
and  enhancements  is  only  the  first  step;  what  is  ultimately  needed  is  an 
assessment  of  the  impact  on  overall  system  performance. 

Quantitative  evaluation  of  proposed  changes  is  required  for  a  number 
of  reasons.  First,  there  is  the  obvious  need  to  know  if  a  proposed  change 
will  actually  improve  the  performance  of  the  overall  system.  Within  a 
complex  system  such  as  WTADS,  instituting  a  change  in  one  area,  or  for 
one  decisionmaker,  may  have  positive  local  effects,  but  may  not  in  fact 
improve  overall  system  performance  because  of  other  constraints  or 
bottlenecks  within  the  system.  Evaluation  is  also  necessary  when  several 
potential  enhancements  are  being  considered  and  the  overall  effective¬ 
ness  of  the  individual  changes  must  be  ranked  or  the  interrelationships 
of  several  simultaneous  changes  understood.  Finally,  the  quantitative 
evaluation  of  proposed  changes  is  often  necessary  to  justify  the  develop¬ 
ment  costs  associated  with  decision  support  tools. 

Evaluating  the  effect  of  changes  on  the  combat  capability  of  sup¬ 
ported  units  is  difficult  for  several  reasons.  The  presence  of  variability 
and  uncertainty  in  a  dynamic  wartime  environment  complicates  any 
evaluation.  Understanding  and  representing  the  variability  and  uncer¬ 
tainty,  combined  with  the  overall  complexity  of  the  ammunition  distri¬ 
bution  system,  makes  subjective  evaluation  almost  impossible.  The 
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alternative  of  using  field  tests  or  exercises  to  evaluate  changes  has 
numerous  shortcomings.  Again,  the  uncertainty  and  complexity  compli¬ 
cate  such  field  tests,  making  realistic  exercises  difficult  to  design  and 
expensive  to  conduct.  When  field  tests  are  used,  the  resulting  evalua¬ 
tions  are  often  limited  because  of  the  need  to  constrain  or  structure 
various  aspects  of  the  overall  exercise.  Typically,  the  only  viable  alter¬ 
native  for  evaluating  potential  changes  is  to  develop  simulation  models 
of  the  overall  environment. 

There  are  a  number  of  simulation  models  that  treat  all  or  part  of  the 
WTADS.  These  models  share  the  following  shortcomings  in  terms  of 
evaluating  the  combat  impact  of  proposed  changes  to  the  WTADS; 

•  The  models  are  “rigid”  in  the  sense  that  they  are  incapable  of 
modeling  changes  that  are  much  different  from  current  doc¬ 
trine. 

•  The  interaction  of  management  and  communication  with  the 
physical  system  cannot  be  studied  because  materiel  manage¬ 
ment  and  movement  control  centers  are  not  played. 

•  The  impact  on  combat  capability  cannot  be  assessed  because 
combat  performance  measures  are  either  entirely  missing  or  are 
of  marginal  relevance. 

•  The  impact  of  uncertainty  cannot  be  evaluated  because  the 
models  are  deterministic. 

•  Several  of  the  models  run  so  slowly  that  studies  of  more  than  a 
handful  of  changes  would  be  prohibitively  expensive. 

The  shortcomings  of  existing  models  and  the  objectives  of  our  research 
suggested  that  a  new  evaluation  tool  should  be  developed. 


PHILOSOPHY  USED  IN  DEVELOPING  THE  MODEL 

From  the  perspective  of  evaluating  the  impact  of  improved  informa¬ 
tion  and  decision  support  tools,  it  is  imperative  that  the  model  of  the 
ammunition  distribution  system  emphasize  the  information  flows  within 
the  overall  system  and  the  decisionmaking  characteristics  of  WTADS 
managers.  In  constructing  the  WTADS  simulation,  we  explicitly  model 
Materiel  Management  Centers  (MMCs),  Movement  Control  Centers 
(MCCs),  and  various  other  ammunition  managers,  such  as  the  Division 
Ammunition  Officer  (DAO).  This  inclusion  of  the  management  aspects 
and  interrelationships  within  the  overall  system  is  one  characteristic 
that  makes  the  WTADS  simulation  model  different  from  other  avail¬ 
able  system  models. 

We  model  within  the  WTADS  simulation  aspects  of  the  entire 
ammunition  distribution  system  by  representing  the  flow  of  ammuni- 


tion  from  CONUS  production  plants  to  the  actual  combat  user.  At 
this  stage  of  model  development,  we  are  focusing  primarily  on  the 
theater  distribution  system  with  only  a  rudimentary  representation  of 
the  ammunition  flow  from  the  plants  to  the  theater.  This  structure  is 
contained  within  the  WTADS  model,  however,  to  enhance  the  first  leg 
of  the  overall  distribution  system  during  a  later  model  iteration. 

An  important  issue  in  the  development  of  the  WTADS  simulation 
was  the  representation  of  the  variability  and  uncertainty  that  will  exist 
in  wartime.  Wartime  variability  and  uncertainty  are  incorporated 
within  the  model  in  a  number  of  ways.  The  model  generates  daily  pos¬ 
tures  for  the  combat  units  included  in  the  simulation.  These  postures 
include  offensive  and  defensive  engagements  at  various  levels  of  inten¬ 
sity.^  Each  posture  relates  to  an  assumed  enemy  action  and  has  associ¬ 
ated  with  it  a  set  of  enemy  targets.  The  model  incorporates  the  uncer¬ 
tainty  and  variability  of  war  by  generating  actual  enemy  actions  and 
target  sets  that  may  differ  from  the  planned  postures  and  target  sets. 
This  allows  the  model  to  measure  how  robust  the  overall  system  and 
proposed  enhancements  are  with  respect  to  unpredictable  enemy  inten¬ 
tions.  The  model  also  incorporates  uncertainty  by  allowing  the  enemy 
to  interdict  and  destroy  various  elements  of  the  distribution  system 
such  as  storage  sites,  material  handling  equipment,  trucks,  and  person¬ 
nel. 

Two  final  characteristics  included  in  our  modeling  philosophy  are 
the  measures  of  performance  calculated  by  the  model  and  the  user- 
related  aspects  of  the  overall  modeling  environment.  Although  the 
model  generates  the  standard  types  of  logistics-oriented  metrics  such  as 
tons  delivered  or  equipment  utilization,  we  felt  that  it  was  also  impor¬ 
tant  to  include  combat-oriented  measures  of  effectiveness.  The  model 
evaluates  the  impact  of  proposed  system  changes  on  combat  capability 
by  calculating  the  number  and  type  of  targets  killed. 

To  provide  a  suitable  modeling  environment  from  a  user  perspective, 
we  developed  the  WTADS  simulation  using  an  object-oriented 
language.  The  use  of  this  construct,  described  in  more  detail  below, 
facilitated  the  overall  modeling  effort  and  results  in  a  simulation  that 
is  easy  for  a  user  to  understand  and  modify. 

Incorporating  the  above  mentioned  points  provides  a  model  that  can 
address  the  evaluation  of  both  structural  changes  to  the  WTADS  and 
enhancements  in  the  information  flow  and  decisionmaking  capabilities 
within  the  WTADS  It  can  also  perform  these  evaluations  using  both 

'Altogether  there  are  nine  postures  that  a  particular  unit  can  assume,  including  the 
various  permutations  of  protracted  attack  and  defense  in  combination  with  intensities  of 
light,  medium,  and  heavy. 
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logistics  and  combat -oriented  measures  of  effectiveness.  An  overview 
of  the  WTADS  model  is  given  next. 


OVERVIEW  OF  THE  WTADS  MODEL 

The  WTADS  model  is  a  simulation  written  in  the  ROSS  language.^ 
ROSS  is  a  language  that  uses  interacting  objects  to  represent  an  overall 
system.  A  simulation  of  a  military  engagement  might  include  objects 
representing  Red  and  Blue  maneuver  units,  physical  resources,  such  as 
trucks  or  weapon  systems,  and  command  and  control  organizations. 
Each  object  has  various  attributes  that  describe  it,  plus  a  set  of 
behaviors  that  enumerate  messages  the  object  may  receive  from  other 
objects  and  messages  that  an  object  may  send  in  response.  ROSS  pro¬ 
vides  a  powerful  and  flexible  environment  in  which  to  build  and  modify 
simulation  models.  Its  English-like  syntax  permits  fairly  rapid  under¬ 
standing  of  the  interactions  contained  within  the  simulation  model, 
even  by  individuals  not  familiar  with  the  language. 

The  WTADS  model  is  composed  of  four  modules  as  shown  in  Fig.  1. 
The  heart  of  the  model  is  the  combination  of  the  physical  distribution 
module  and  the  management  decision  module.  These  modules  contain 
the  attributes,  behaviors,  and  messages  for  the  numerous  objects  con¬ 
tained  in  the  overall  simulation.  These  objects,  or  actors,  include  com¬ 
bat  and  support  units,  storage  locations,  trucks,  roads,  and  various 
decisionmaking  managers  or  organizations  from  the  division  to  the 
theater  Army  level.  The  physical  distribution  module  passes  resource 
status  information  to  the  management  decision  module,  which  in  turn 
passes  back  courses  of  actions  (e.g.,  what  ammunition  to  ship  where,  at 
what  time,  and  using  which  resources)  based  on  the  behaviors  and 
rules  representing  the  managers’  decision  strategies.  The  physical  dis¬ 
tribution  module  then  executes  the  directives  and  passes  back  the 
resulting  resource  status  information. 

Interacting  with  these  two  modules  are  the  scenario  generator  and 
the  combat  evaluator  modules.  The  scenario  generator  develops  pos¬ 
tures  (operational  orders)  for  the  various  units,  targets  associated  with 
these  postures,  and  data  on  losses  due  to  enemy  actions.  The  physical 
distribution  module  incorporates  the  damage  reports  into  the  resource 
status  information,  and  the  management  decision  module  uses  the 
operational  orders  implied  by  the  postures  when  determining  courses  of 
action.  The  last  module,  the  combat  evaluator,  combines  the  target 
array  faced  by  combat  units  and  the  ammunition  delivered  from  the 

^The  ROSS  language  is  described  in  Ref.  2;  previous  applications  of  ROSS  simulation 
models  are  described  in  Refs.  .3  and  4. 
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Fig.  1 — WTADS  model  components 


distribution  module  into  a  combat-oriented  performance  measure  (such 
as  targets  killed  or  percent  of  “correct”  ammunition  delivered). 


EXAMPLE  OF  USING  THE  WTADS  MODEL 

As  an  initial  demonstration  of  the  WTADS  model,  we  considered 
the  problem  of  scheduling  trucks  for  ammunition  pickup  at  ammuni¬ 
tion  transfer  points  (ATPs).  Trailers  from  the  corps  storage  areas 
deliver  ammunition  to  the  ATPs  where  they  are  picked  up  by  trucks 
from  the  combat  units.  Currently,  the  unit  trucks  arrive  randomly  to 
pick  up  their  loads  from  the  ATP,  although  some  DAOs  attempt  to 
schedule  truck  arrivals  to  minimize  waiting  times  and  congestion. 

The  WTADS  model  was  first  exercised  using  random  arrivals  for  the 
unit  trucks.  This  assumes  that  the  DAO  either  has  no  visibility  of 
when  corp  trailers  will  resupply  the  ATPs  or,  if  he  does  have  trailer 
arrival  information,  does  not  attempt  to  schedule  unit  trucks  into  the 
ATP.  At  the  other  extreme,  we  assume  that  the  DAO  has  perfect 
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forecasts  of  trailer  arrivals  and  schedules  the  unit  trucks  to  minimize 
their  waiting  times  at  the  ATP.  Since  perfect  forecasts  are  highly 
unlikely,  a  more  reasonable  assumption  of  plus  or  minus  two  hours  of 
scheduling  accuracy  was  used  for  a  third  model  run.  Each  of  the  three 
model  runs  involved  fairly  simple  modifications  of  the  appropriate 
parts  of  the  WTADS  model  logic  to  reflect  tne  scheduling  rules. 

The  results  of  the  three  WTADS  runs  examining  the  effect  oi 
scheduling  accuracy  are  shown  in  Figs.  ‘2  and  3.  In  Fig.  2,  the  measure 
of  interest  was  the  number  of  trucks  waiting  at  the  ATP.  The  model 
indicates  that  with  perfect  information  and  scheduling,  the  number  of 
waiting  trucks  can  be  reduced  by  two-thirds  over  the  completely  ran¬ 
dom  scheduling  alternative.  Even  if  trucks  can  be  scheduled  within  a 
two  hour  window,  the  number  of  waiting  trucks  can  be  reduced  by  a 
third  over  the  random  scheduling  alternative.  The  advantages  of  fewer 
tnicks  waiting  at  the  ATP  include  the  availability  of  more  trucks  for 
other  missions,  reduction  in  congestion,  and  reduction  of  target  signa¬ 
ture. 

Figure  3  shows  the  effects  of  scheduling  accuracy  on  the  average 
number  of  tank  equivalents  killed.  Perfectly  coordinated  scheduling  of 
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Fig.  3 — Improved  scheduling  coordination  at  the  ATP: 
Average  number  of  tank  equiva  .  nts  killed 


ammunition  pickup  at  the  ATP  would  increase  tank  kills  by  approxi¬ 
mately  10  percent  over  the  uncoordinated  case. 

The  ability  to  improve  the  scheduling  of  pickup  and  deliveries  at  the 
ATP  is  just  one  example  of  how  the  WTAD.S  model  could  be  used  to 
evaluate  the  effects  of  docision  support  enhancements.  The  current 
phase  of  our  research  uses  the  WTADS  model  to  evaluate  several 
structural  and  doctrinal  changes  currently  being  considered  by  the 
Army.  These  evaluations  are  described  below. 


EXTENSION  OF  PHASE  II  RESEARCH 

The  primary  rationale  for  developing  the  WTADS  simulation  model 
was  to  provide  a  method  for  evaluating  the  combat  effects  of  changes 
in  policy,  procedure,  or  force  structure.  We  will  examine  three  specific 
logistics  infrastructure  and  policy  changes  to  the  ammunition  distribu¬ 
tion  system  during  the  next  phase  of  our  research: 


1 
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•  The  introduction  of  the  palletized  load  system  (PLS) 

•  The  implementation  of  certain  aspects  of  the  maneuver- 
oriented  ammunition  distribution  system  (MOADS) 

•  The  development  of  special  management  procedures  for  high 
lethality,  high  technology  munitions. 

The  PLS  is  basically  a  new  vehicle  that  is  designed  to  pick  up, 
transport,  and  offload  pallets  of  ammunition.  The  objective  of  the  sys¬ 
tem  is  to  reduce  the  requirements  for  trailers  and  material  handling 
personnel  and  to  improve  the  flow  of  ammunition  in  the  theater. 
MOADS  involves  a  number  of  management  initiatives  including  pre- 
conftgured  ammunition  loads  and  changes  to  the  number  and  responsi¬ 
bilities  of  the  various  ammunition  storage  points  within  the  theater. 
High  technologv'  munitions  are  important  because  of  their  potential  to 
overcome  the  quantitative  disadvantages  faced  by  U.S.  forces.  How¬ 
ever,  because  of  their  high  cost,  these  “silver  bullets"  are  typically  in 
short  supply.  Special  management  procedures  for  the  inventory  and 
distribution  of  these  munitions  may  provide  the  responsiveness  and 
flexibility  needed  to  overcome  the  numerical  shortfalls. 

In  addition  to  the  three  specific  changes  described  above,  we  are 
evaluating  a  number  of  management  initiatives  including  changing 
scheduling  operations  at  storage  facilitie.s,  using  various  stockage  objec¬ 
tives  at  ATPs.  and  providing  responsive  reallocation  of  transportation 
between  routes  and  storage  sites. 

To  evaluate  each  of  the  above  changes,  the  WTADS  simulation 
model  will  be  exerci.sed  for  the  current  system  and  for  the  “changed" 
system,  and  the  differences  in  combat  outcomes  measured.  To  incor¬ 
porate  each  of  the  above  changes  inio  the  simulation  model,  the  attri¬ 
butes  and  behaviors  of  various  simulated  objects  will  be  modified  to 
reflect  the  specific  aspects  of  each  change. 

Because  of  the  uncertainty  surrounding  the  wartime  environment,  it 
will  be  necessary  to  evaluate  the  various  changes  for  a  wide  range  of 
scenarios. '  we  must  understand  if  proposed  procedural  and  structural 
changes  improve  combat  performance  for  only  a  narrow  range  of 
potential  courses  of  action  or  if  they  are  more  robust,  improving  overall 
system  performance  for  a  wide  spectrum  of  potential  scenarios. 

As  the  WTADS  model  was  being  developed  during  Phase  II  of  the 
research,  a  .second,  parallel  path  examining  the  potential  of  applying 
expert  system  technology  in  the  WTADS  arena  was  also  pursued.  The 


'Scenario  chanfre.s  will  affect  the  probabilities  of  combat  units  assuming  different  nos 
tores  in  addition  to  changes  in  the  target  density-  Higher  probabilities  of  offensne  ami 
defensive  postures  and  larger  target  sets  imply  a  more  stresstul  combat  environment 
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Phase  II  results  of  the  expert  system  assessment  and  evaluation  and 
the  directions  of  our  Phase  III  research  on  expert  systems  are  described 
in  the  next  section. 


III.  ASSESSMENT  AND  EVALUATION  OF 
EXPERT  SYSTEM  TECHNOLOGY 


A  second  high  leverage  research  area  identified  during  Phase  I  of  the 
project  was  the  further  examination  of  expert  system  technology  to 
assist  WTADS  managers.  The  overall  size  and  complexity  of  the 
WTADS  suggested  that  there  were  numerous  problems  faced  by  the 
system  managers  whose  solutions  might  be  aided  by  the  development 
of  special  purpose  expert  systems.  This  second  research  path  was 
directed  at  developing  a  list  of  potential  applications  for  expert  systems 
and  a  methodology  for  screening  and  ranking  the  problem  applications 
to  determine  which  were  the  best  candidates  for  prototype  expert  sys¬ 
tem  development. 

This  section  describes  the  assessment  and  evaluation  of  expert  sys¬ 
tem  technology  accomplished  during  Phase  II  and  outlines  current 
analysis  in  this  area.  First,  we  present  the  rationale  for  using  expert 
systems  for  WTALS  decisionmaking  problems.  We  then  summarize 
the  screening  mechanism  and  describe  the  results  of  applying  it 
mechanism  to  a  candidate  list  of  problems  faced  by  WTADS  decision¬ 
makers.  We  conclude  the  section  by  discussing  one  hurdle— the  lack  of 
wartime  expertise — that  must  be  overcome  during  the  expert  system 
development  process  and  describe  the  ways  we  plan  to  deal  with  this 
problem  during  Phase  III  of  our  research. 


THE  RATIONALE  FOR  EXPERT  SYSTEMS 
FOR  WTADS  TASKS 

The  overall  wartime  theater  ammunition  distribution  system  will  be 
very  large  and  complex,  involvi^ig  numerous  decisionmakers  interacting 
to  ensure  ammunition  flows  from  the  ports  to  the  combat  users.  Deci¬ 
sions  must  be  made  about  what  ammunition  to  distribute,  when  to  dis¬ 
tribute  it,  where  to  send  it,  and  how  to  store,  handle,  and  transport  it. 
Each  of  these  decisions  involves  a  large  number  of  alternatives  and  fac¬ 
tors,  resulting  in  many  possible  solutions  to  each  individual  question. 
From  this  large  solution  space,  the  system  managers  must  choose  effec¬ 
tive  solutions. 

The  wartime  ammunition  distribution  system  managers’  environ¬ 
ment  will  be  highly  dynamic  and  stressful.  The  managers  will  have  to 
make  decisions  within  a  short  timeframe;  they  may  not  have  available 
all  the  information  they  need  to  make  a  “good”  decision;  the 
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information  that  is  available  may  be  corrupted  or  incorrect;  and  the 
consequences  of  a  “bad”  decision  may  be  catastrophic. 

Adding  to  the  complications  caused  by  the  complex  and  stressful 
nature  of  the  wartime  environment,  many  distribution  managers  will 
be  faced  with  decisions  for  which  they  have  had  little  training  and 
experience.  Probably  more  so  than  other  military  “systems.”  the 
ammunition  distribution  system  operates  very  differently  in  peacetime 
than  it  will  in  wartime.  By  its  very  nature,  little  ammunition  is 
expended  in  peacetime,  certainly  nowhere  near  the  quantities  that 
must  be  moved  in  wartime.  Thus,  there  are  few  highly  trained  ammu¬ 
nition  distribution  managers. 

The  characteristics  of  the  wartime  environment  and  the  small 
number  of  experienced  WTADS  managers  suggest  that  special  purpose 
expert  systems  could  benefit  decisionmakers.  The  inference  mechan- 
iarfis  of  expert  systems  permit  structured  searches  through  large  solu¬ 
tion  spaces  to  find  appropriate  solutions  to  problems.  Expert  systems 
could  assist  human  decisionmakers  in  collating  the  numerous  factors 
involved  in  the  WTADS  decision  process  and  in  evaluating  the  various 
alternatives.  Expert  systems  could  have  explanation  procedures  to  help 
WTADS  decisionmakers  understand  why  specific  alternatives  are  pre¬ 
ferred  or  why  other  alternatives  do  not  offer  feasible  solutions.  Finally, 
capturing  within  an  expert  system  the  limited  expertise  that  is  avail¬ 
able  would  be  one  way  of  distributing  that  expertise  throughout  the 
overall  WTADS. 

Special  purpose  expert  systems  may  interface  with  other  decision 
support  technologies,  such  as  traditional  operations  research  tools,  to 
assist  WTADS  decisionmakers.  For  certain  decision  tasks,  operations 
research  techniques  or  improved  information  processing  capabilities 
may  be  more  appropriate  than  expert  systems.  What  is  required  is  a 
mechanism  to  sort  through  the  numerous  tasks  faced  by  WTADS 
managers  and  locate  those  tasks  that  offer  a  high  payoff  for  the 
development  of  expert  decision  aids.  Specifically,  a  problem  enumera¬ 
tion  and  expert  system  screening  technique  is  needed  to  identify  those 
decision  problems  faced  by  WTADS  managers  that  would  be  amenable 
to  expert  system  solutions  and  to  determine  which  of  those  potential 
expert  systems  could  and  should  be  developed.  An  expert  system 
screening  mechanism  is  described  next. 


MATCHING  TASKS  TO  EXPERT  SYSTEM  SOLUTIONS 

An  expert  system  screening  procedure  developed  at  The  RAND  Cor¬ 
poration  and  described  by  Waterman  (Ref.  5)  was  extended  and  applied 
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to  determine  the  relative  merit  of  developing  an  expert  system  for  a 
particular  WTADS  decisionmaking  task.  The  screening  procedure 
includes  questions  on  whether  a  problem  is  appropriate  for  an  expert 
system,  whethp*'  an  '"xpert  ovotem  :?  fcfic'tlc,  and  whether  en  expert 
system  development  effort  is  justified. 

Each  of  these  three  categories  involves  a  number  of  finer,  more 
detailed  questions.  For  example,  appropriateness  includes  whether  or 
not  the  problem  is  of  manageable  size,  whether  it  involves  a  suitable 
level  of  difficulty,  and  whether  the  solution  process  is  primarily  heuris¬ 
tic  versus  algorithmic,  as  well  as  several  other  similar  concerns.  Feasi¬ 
bility  includes  questions  involving  the  availability  of  experts,  the  ability 
of  experts  to  articulate  their  solution  process,  and  whether  there  is  a 
consensus  among  the  experts  on  the  solution  procedures.  Similarly, 
the  development  of  an  expert  system  is  justifiable  if  expertise  is  scarce, 
if  available  expertise  is  being  lost,  or  if  the  expertise  is  required  at 
several  locations.  Figure  4  shows  one  of  the  worksheets  used  during 
the  screening  process. 

In  addition  to  identifying  problems  that  are  appropriate  for  expert 
system  development,  the  screening  mechanism  identifies  other 
appropriate  solution  technologies,  such  as  operations  research  methods 
or  manual  procedures,  that  might  aid  WTADS  decisionmakers.  For 
some  tasks,  the  screening  process  might  indicate  that  no  change  in  the 
solution  procedures  is  warranted  or  that  a  combination  of  expert  system 
and  other  technologies  are  appropriate. 

Many  of  the  specific  criteria  contained  in  the  screening  mechanism 
involve  subjective,  qualitative  evaluations  by  someone  knowledgeable  in 
the  policies  and  procedures  of  ammunition  distribution.  The  screening 
procedure  proposed  by  Waterman  in  essence  provided  guidelines;  the 
application  of  those  guidelines  was  not  a  simple  task  and  involved 
numerous  iterations.  To  demonstrate  the  application  of  the  procedure 
and  to  identify  WTADS  tasks  that  might  be  aided  by  expert  systems,  a 
knowledge  engineer  and  an  ammunition  distribution  domain  expert' 
worked  closely  through  several  interactive  question  and  answer  ses¬ 
sions.  The  WTADS  tasks  that  were  identified  and  the  results  of  pass¬ 
ing  those  tasks  through  the  screening  procedure  are  shown  in  Fig.  5. 

Although  many  (eight  of  the  fourteen)  decisionmaking  problems 
were  judged  appropriate  for  expert  systems  and  the  development  of 
those  systems  was  justifiable,  most  problems  fell  short  of  the  feasibility 
criterion.  A  serious  problem  facing  the  development  of  expert  systems 
is  the  apparent  lack  of  WTADS  managers  with  suitable  wartime 

'LTC  Sam  Cantey,  an  Army  fellow  assigned  to  The  RAND  Corporation,  has  exten¬ 
sive  background  in  the  ammunition  arena  and  served  as  the  domain  expert  during  this 
phase.  LTC  Jim  Price  is  our  current  “expert.” 
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Fig.  5 — Results  of  applying  the  expert  system  screening  procedure 


19 


expertise.^  This  is  also  true  for  other  areas  of  the  Combat  Service  Sup¬ 
port  system  in  wartime,  notably  the  management  of  Class  IX  (repair 
parts)  and  Class  III  (fuel). 

The  general  lack  of  domain  experts  should  not  be  surprising.  Policy 
and  force  structure  changes  have  been  implemented  in  the  last  several 
years  that  have  affected  ammunition  distribution  system  management. 
WTADS  managers  have  not  had  much  opportunity  to  react  to  these 
changes  and  develop  new  heuristic  procedures  required  to  face  the 
problems  arising  from  the  uncertainty  of  wartime  and  the  complexity 
of  the  overall  system.  Even  peacetime  field  exercises  and  wargames  do 
not  provide  the  “realism”  necessary  to  develop  expertise  in  making 
decisions  and  managing  resources.  The  necessary  heuristic  rules  can 
only  result  from  managers  repeatedly  facing  unexpected  problems  and 
seeing  the  results  of  their  decisions. 


PHASE  III  RESEARCH  OBJECTIVES 

The  results  of  the  second  phase  of  the  research  on  the  application  of 
expert  system  technology  include  the  adaptation  and  application  of  a 
procedure  to  screen  tasks  in  order  to  identify  those  that  might  benefit 
from  the  development  of  expert  systems  and  the  identification  of  a 
potentially  serious  hurdle  for  the  development  of  expert  systems— the 
lack  of  available  wartime  expertise.  The  objectives  of  our  research  on 
expert  systems  are  to  demonstrate  the  technology  by  developing  a  pro¬ 
totype  expert  system  for  a  problem  where  there  is  an  apparent  lack  of 
expertise  and  to  explore  methods  for  extending  this  prototype  system 
by  cultivating  new  knowledge  and  expertise. 


Development  of  a  Prototype  Expert  System 

Identifying  problems  that  appear  appropriate  for  the  development  of 
expert  systems  is  only  the  first  step;  actually  developing  the  expert  sys¬ 
tem  and  then  evaluating  its  impact  on  system  performance  must  also 
be  accomplished.^ 

^Twelve  of  the  fourteen  tasks  failed  the  feasibility  criterion  because  of  the  lack  of 
suitable  expertise.  Only  the  management  of  storage  site  configurations  and  the  layout  of 
storage  sites  passed  the  expertise  experience  filter. 

^The  term  “evaluation”  carries  different  connotations.  Technical  evaluation  addresses 
whether  the  expert  system  properly  represents  the  solution  procedures,  subjective  evalua¬ 
tion  is  concerned  with  how  the  users  perceive  the  strengths  and  weaknesses  of  the  sys¬ 
tem,  and  empirical  evaluation  measures  how  well  the  system  improves  overall  perfor¬ 
mance.  We  are  concerned  here  with  the  empirical  evaluation  of  expert  systems. 
Methods  for  accomplishing  the  evaluations  are  discussed  in  the  next  section. 


Our  current  research  is  examining  the  problem  of  developing  expert 
Sj  stems  when  there  is  a  lack  of  expertise  and  experience.  Two 
approaches  to  solving  this  problem  include  (1)  investigating  methods  to 
extend  existing  WTADS  domain  knowledge  and  then  cultivate  the 
expertise  when  building  an  expert  system,  and  (2)  examining  the  tech¬ 
nology  of  systems  that  can  “learn”  from  actual  field  use  and  adapt 
their  behavior  based  on  this  learning. 

To  focus  our  research  efforts,  we  have  constructed  a  “zero-base” 
expert  system  for  the  problem  of  relocating  ammunition  transfer 
points.  This  problem  includes  when  to  relocate  an  ATP,  where  to  relo¬ 
cate  it  to,  and  how  to  accomplish  the  relocation.  The  ATP  relocation 
problem  is  important  because  of  the  conflicting  tradeoffs  between  the 
desire  to  be  close  to  the  combat  users  yet  be  sufficiently  removed  from 
hostilities  to  reduce  the  probability  of  enemy  attack.  Our  view  is  that 
current  WTADS  managers  do  not  understand  the  implications  of  these 
tradeoffs  very  well.  The  existence  of  an  expert  system  for  this  prob¬ 
lem,  coupled  with  the  ability  to  evaluate  through  the  WTADS  model 
the  combat  effects  of  alternative  decisions  and  policies,  would  help 
WTADS  managers  develop  the  needed  expertise. 

This  initial  system  was  developed  based  on  the  expertise  that  does 
exist  and  on  published  doctrine.  Our  research  is  now  addressing  the 
problem  of  lack  of  experience  by  investigating  both  methods  to  cul¬ 
tivate  management  expertise  and  systems  that  can  adapt  and  respond 
to  the  uncertainties  that  arise  in  wartime. 

Extending  and  Cultivating  Expertise 

Suitable  levels  of  expertise  are  lacking  among  WTADS  managers 
because  the  majority  of  system  managers  have  had  little  or  no  exposure 
to  the  uncertain  and  dynamic  environment  of  wartime.  The  con¬ 
strained,  limited  types  of  functions  performed  in  peacetime  do  not 
allow  the  ammunition  distribution  system  decisionmakers  to  see  all  of 
the  types  of  problems  that  will  arise  in  wartime  or  to  understand  the 
implications  of  their  decisions.  Stated  doctrine  can  only  take  the 
WTADS  managers  so  far;  experience  in  wartime  decisionmaking  is 
required  to  assist  them  in  developing  heuristic  procedures  for  dealing 
with  the  problems  that  fall  outside  of  doctrine. 

One  way  to  assist  WTADS  managers  in  developing  heuristic  pro¬ 
cedures  is  to  simulate  the  uncertainties  of  wartime  and  present  the 
decisionmakers  with  representations  of  re.^.listic  problems.  This  can  be 
accomplished  with  the  WTADS  model  by  replacing  one  of  the  objects 
in  the  simulation  model  with  a  human  decisionmaker.  Through  this 
human-model  interaction,  we  can  represent  an  environment  where  an 
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ammunition  distribution  manager  can  see  the  combat  effects  of  his 
decisions.  With  repeated  trials  of  similar  decision  problems,  supple¬ 
mented  by  feedback  mechanisms,  the  decisionmaker  can  develop 
heuristic  procedures  to  address  the  problems  that  arise  in  the  uncertain 
environment  of  wartime.  That  is,  the  WTADS  decisionmakers  can 
extend  their  knowledge  base.  Based  on  the  results  of  these  experi¬ 
ments,  the  initial  expert  system  can  be  refined.  The  laboratory 
environment,  where  a  human  can  work  with  the  simulation  model  for 
expertise  cultivation  and  training,  is  an  important  aspect  of  our  future 
research  and  will  be  discussed  in  more  detail  later. 


Adaptive  Expert  Systems 

A  second  technique  for  overcoming  the  lack  of  wartime  expertise  is 
to  examine  the  viability  of  systems  that  can  extend  their  knowledge  by 
“learning”  based  on  the  data  they  receive  during  actual  use.  Adaptive 
systems  have  been  of  interest  in  the  artificial  intelligence  community 
for  some  time  and  research  in  this  area,  although  primarily  theoretical, 
has  yielded  some  commercial  applications  such  as  computer  programs 
that  can  induce  rules  based  on  sets  of  examples.  The  ability  to  develop 
adaptive  expert  systems  can  result  in  decision  aids  that  can  function  in 
situations  that  cannot  be  represented  in  a  simulated  environment. 

In  this  section  and  the  previous  one,  we  have  described  the  results  of 
the  two  parallel  research  paths  pursued  during  Phase  II  and  how  we 
are  extending  these  research  paths  during  Phase  III.  In  the  next  sec¬ 
tion,  we  will  describe  several  new  research  objectives  that  use  the 
WTADS  model  for  evaluative  purposes. 


IV.  EVALUATION  AND  IMPLEMENTATION  OF 
WTADS  DECISION  AIDS 


The  WTADS  simulation  model  was  developed  to  provide  a  mecha¬ 
nism  for  evaluating  changes  to  the  wartime  ammunition  distribution 
system.  Changes  to  the  WTADS  could  be  of  a  doctrinal  or  policy 
nature,  such  as  the  number  and  composition  of  the  various  storage 
locations,  or  could  involve  changes  to  the  logistics  infrastructure,  such 
as  introducing  new  material  handling  equipment.  The  evaluation  of 
PLS  and  MOADS  and  the  intensive  management  of  high  technology 
munitions  fall  into  these  categories.  The  main  thrust  of  our  research, 
however,  is  identifying  and  evaluating  changes  to  the  information  pro¬ 
cessing  and  the  decisionmaking  aspects  of  the  WTADS.  The  WTADS 
model  will  also  be  used  to  evaluate  these  information  management 
types  of  changes. 

In  addition  to  pursuing  the  two  research  paths  initiated  during  the 
second  phase  of  our  research,  our  Phase  III  analysis  encompasses  three 
new  research  objectives: 

•  Predevelopment  evaluation  of  proposed  decision  aids 

•  Postdevelopment  evaluation  of  prototype  decision  aids 

•  Examination  of  methods  to  implement  decision  aids. 

This  section  describes  the  rationale  for  pursuing  each  of  the  three 
objectives  and  briefly  outlines  the  research  approach.  Similar  to  the 
problem  of  cultivating  expertise  among  WTADS  decisionmakers, 
research  on  the  evaluation  and  implementation  of  decision  aids  will 
require  interfacing  human  decisionmakers  with  the  WTADS  simulation 
model.  To  provide  a  realistic  environment  to  support  our  various 
research  objectives,  we  foresee  the  need  to  develop  a  WTADS  labora¬ 
tory  in  which  to  integrate  decisionmakers  and  the  WTADS  model  for 
observation  and  analysis.  The  section  concludes  by  describing  the  con¬ 
cepts,  operations,  and  benefits  of  a  WTADS  laboratory. 


EVALUATION  OF  DECISION  AIDS 

The  term  evaluation  of  decision  aids  has  different  meanings  depend¬ 
ing  on  the  context  in  which  it  used.  Commonly,  evaluation  of  decision 
aids  refers  to  a  technical  evaluation,  or  validation,  of  whether  the  deci¬ 
sion  aid  correctly  performs  its  intended  functions.  For  example,  with 


22 


expert  systems  this  type  of  evaluation  examines  whether  the  expert 
system  replicates  the  processes  and  results  of  a  human  “expert”  when 
faced  with  similar  problems  and  information  on  those  problems. 
Technical  evaluation  is  usually  performed  by  people  knowledgeable  in 
the  domain  comparing  the  results  of  the  expert  system  with  the  results 
of  human  experts,  or  with  the  actual  “answer,”  for  a  range  of  appropri¬ 
ate  case  studies.  Technical  evaluation  is  necessary  for  any  decision 
aid  or  model,  but  is  not  the  focus  of  our  research. 

Evaluation  can  also  refer  to  the  subjective  evaluation  of  the  strengths 
and  weaknesses  of  the  decision  aid  when  supporting  a  human  to  solve 
problems.  This  type  of  evaluation  examines  the  interfaces,  displays, 
input  and  output  requirements,  timeliness,  and  other  aspects  of  the 
interface  between  the  decision  support  system  and  the  human  decision¬ 
maker.  Subjective  evaluations  are  often  performed  by  having  a  range 
of  decisionmakers  use  the  system  and  then  provide  comments  on  a 
questionnaire.  Subjective  evaluations  are  also  necessary  and  important 
to  ensure  that  decisionmakers  will  use  the  aid  and  benefit  from  its 
availability,  and  we  will  make  subjective  evaluations  when  investigating 
ways  to  implement  decision  aids  within  the  wartime  ammunition  distri¬ 
bution  system. 

We  will  typically  use  the  term  evaluation  to  refer  to  the  empirical 
evaluation  of  whether  the  decision  aid  has  an  effect  on  overall  system 
performance.  An  aid  may  be  technically  sound  and  may  have  efficient 
and  effective  interfaces  with  users  but  may  not  actually  improve  the 
overall  decisionmaking  capabilities.  This  type  of  evaluation  is  often 
neglected  during  the  development  and  implementation  of  decision  aids, 
usually  because  the  effect  on  performance  is  difficult  to  measure, 
especially  for  complex  environments  such  as  the  wartime  theater 
ammunition  distribution  system.  We  intend  to  use  the  WTADS  model 
for  empirical  evaluations  of  expert  systems  and  decision  aids  in  gen¬ 
eral,  both  after  a  decision  aid  is  developed  and  before  development 
efforts  have  begun. 


PREDEVELOPMENT  EVALUATIONS 

Most  evaluations  of  decision  aids  occur  after  a  prototype  or  initial 
version  of  the  software  model  is  available.  These  evaluations  typically 
address  the  technical  “correctness”  of  the  decision  aid  or  the  user  per¬ 
ceptions  of  the  interactions  with  the  tool.  Such  evaluations  are  impor¬ 
tant  to  understand  what,  if  any,  improvements  are  necessary  before  a 
decision  tool  reaches  the  implementation  stage  of  development.  Rarely 
does  the  evaluation  of  the  prototype  address  the  overall  impact  on 
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system  performance  and  never,  to  the  I  est  of  our  knowledge,  are  such 
empirical  evaluations  conducted  before  development  efforts  commence. 

The  ability  to  conduct  predevelopment  evaluations  is  necessary  for  a 
number  of  reasons.  The  design  and  development  of  decisioji  tools  are 
often  expensive  undertakings  that  require  significant  commitmeni.-;  of 
time  and  resources.  The  predevelopment  evaluation  of  the  effect  on 
performance  of  a  proposed  decision  tool  would  help  in  justifying 
development  efforts.  There  have  been  efforts  in  information  manage¬ 
ment  to  construct  methods  to  estimate  the  cost  of  developing  decis^in 
support  or  other  types  of  sc.^waic  systems.  These  efforts  permit  the 
examination  of  one  aspect  of  a  cost/benefit  analysis  of  proposed 
software  enhancements.  The  piedevelopment  evaluations  that  we  are 
addressing  would  examine  the  benefit  side. 

Cost/benefit  analyses  play  an  important  part  in  choosing  between 
several  proposed  system  improvements,  especij.lly  when  various  weapon 
systems  are  under  consideration.  Cost  estimates  and  weapon  system 
effectiveness  calculations  have  been  developed  and  used  to  address  the 
costs  and  benotits  of  proposed  hardware  systems.  Because  similar 
“effectiveness”  techniques  have  not  been  developed  for  software 
enhancements,  proposed  decision  tools  often  suffer  when  they  must 
compete  with  hardware  for  limited  development  resources.  Predevelop¬ 
ment  evaluations  of  proposed  decision  aids  would  greatly  hel^  in  their 
competition  with  weapons  for  the  funding  available. 

Finally,  the  ability  to  make  predevelopment  evaluations  of  decision 
tools  is  important  when  several  candidate  aids  are  proposed,  but  only  a 
subset  of  them  can  be  developed  because  of  limited  resources.  Within 
a  large  complex  system  such  as  the  theater  ammunition  distribution 
system,  there  are  numerous  decisionmakers,  each  with  unique  prob¬ 
lems,  at  the  various  nodes  of  the  system.  One  important  research 
question  concerns  identifying  those  decisionmaking  problems  where 
improved  decision  support  tools  would  provide  the  highest  leverage  on 
overall  system  performance.  That  is,  if  a  large  number  of  potential 
management  decision  aids  (expert  systems,  decision  support  systems, 
or  even  manual  aids)  can  be  identified  for  possible  development,  we 
will  consider  how  they  could  be  ranked  in  terms  of  their  impact  on  sys¬ 
tem  performance. 

The  approach  we  are  adopting  for  predevdopment  evaluations  is  to 
first  run  the  WTADS  model  with  current  decisionmaking  capability 
and  measure  the  logistics  and  combat  outcomes.  We  then  modify  the 
WTADS  model  to  reflect  the  local  impact  of  a  proposed  decision  aid 
and  measure  the  changes  in  both  logistics  and  combat  performance. 
Measuring  these  changes  for  a  number  of  proposed  decision  aids  will 
allow  an  initial  ranking  of  the  benefits  of  the  various  propo.sed 
enliancements. 


Although  the  appro;'-,!'  outlined  above  is  conceptually  not  difficult, 
there  are  a  number  of  t«>.rini<  al  challenges.  A  primary  concern  is  how 
to  represent  the  effects  of  jroposed  aecision  aids  in  the  WTADS 
model.  Our  initial  impressio.'i  is  that  the  representation  of  tne  capabil 
ities  of  an  aid  will  he  somewhat  easier  for  certain  classes  or  types  of 
decision  problems  than  for  other  .  For  example,  in  Sec.  II  we  gave  an 
t-tample  of  using  the  model  to  e.vamine  the  effects  of  improved  schedul¬ 
ing  of  ammunitic  n  pickups  .1  an  ATP  (Figs.  2  and  3).  The  object- 
oriented  structure  of  the  R'  S  language  allowed  us  to  represent  dif 
ferent  scheduling  accurad  in'  changing,  he  messages  that  are  sent  or 
tl,  responses  to  messat^'s  that  are  received.  Other  types  of  decision 
proti'c’Tis  c  m  be  represv'e''  "v  changing  the  appropriate  messages  and 
responses,  tu'wever  some  types  of  decision  enhancements  may  be 
more  difficult  to  represent.  In  s'”  n  cases,  other  nodels  that  can  more 
suitably  represent  the  effect  of  the  propeted  decision  aid  can  be  inter¬ 
faced  with  the  WTADS  model  to  understand  the  eject  on  combat  per¬ 
formance. 

A  second  technical  question  involves  determining  the  capability  of  a 
decision  tool  before  the  tool  is  developed.  For  example,  we  may  be  able 
to  represent  “perfect”  decisionmaking,  as  we  did  with  the  example  of 
scheduling  pickups  at  .■\TPs.  But,  how  do  we  know  if  a  tool  could 
actually  achieve  that  level  of  capability?  One  way  is  to  perform  seiisi- 
tivi  y  analyses  on  the  capability  of  a  proposed  decision  tool.  For  exam¬ 
ple,  we  backed  off  from  perfect  scheduling  capability  in  our  earlier 
e'"tmple  to  a  scheduling  capability  of  plus  or  minus  two  hours.  By  per¬ 
forming  a  number  of  the.se  sensitivity  runs  on  the  capability  of  the 
decision  aid,  we  can  draw  curves  similar  to  those  shown  in  Fig.  6. 

Curves  showing  the  tradeoff  between  decisionmaking  capability  and 
system  performance  allow  one  to  assess  the  advantages  .>f  increased 
capability.  For  example,  curve  B  of  Fig.  6  suggests  that  there  is  a 
linear  relationship  between  the  capability  of  i  decision  aid  and  *he  per¬ 
formance  of  the  sysiem.  Therefore,  if  we  could  develop  a  decision  aid 
that  is  10  percent  more  capable  than  current  procedures,  we  could 
increase  the  performance  of  the  system  by  a  corresponding  propor¬ 
tional  amount.  The  other  two  curves  of  Fig.  6  provide  other  types  of 
information.  In  curve  C,  for  exampD,  a  slight  improvement  in 
decisionmaking  capability  from  the  present  position  rtould  result  in  a 
la-ger  increase  in  system  performance;  curve  A  suggests  the  opposite 
effect.  Curves  showing  the  effect  on  .system  performance  of  increased 
decisionmaking  capability  can  help  identify  tho.se  decision  aids  that 
have  high  leverage  impacts.  They  can  also  be  used,  in  combination 
with  subjective  measures  of  the  potential  capability  of  a  decis.  n  aid,  ^o 
estimate  the  overall  impact  on  system  performance  (i.e.,  to  determine  a 
specific  point  on  the  curve). 
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Fig.  6— Decision  aid  capability  vs.  system  performance  curves 


A  final  technical  concern  involves  the  robustness  of  a  proposed  deci¬ 
sion  aid.  For  example,  a  decision  aid  may  have  little  impact  on  perfor¬ 
mance  for  scenarios  that  do  not  result  in  severe  stress  on  the  overall 
system  but  may  have  significant  effects  for  those  types  of  engagements 
that  require  large  quantities  of  ammunition  to  be  distributed.  To 
examine  the  robustness  of  a  proposed  decision  aid,  we  will  make  the 
predevelopment  evaluations  for  a  range  of  scenarios.  We  could  then 
draw  curves,  similar  to  those  in  Fig.  6,  that  show  the  impact  on  perfor¬ 
mance  of  a  decision  aid  under  various  degrees  of  system  “stress." 


POSTDEVELOPMENT  EVALUATIONS 

A  second  research  question  we  are  addressing  is  the  evaluation  of  a 
decision  aid  after  a  prototype  has  been  built.  We  again  use  the  term 
“evaluation”  to  refer  to  the  effect  of  the  decision  aid  on  overall  system 
performance.  This  postdevelopment  evaluation  is  necessary  to  deter¬ 
mine  if  a  prototype  tool  has  sufficient  value  to  justify  further  develop¬ 
ment  efforts  and  ultimately  be  implemented  in  the  field. 
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As  has  been  mentioned  previously,  most  postdevelopment  evalua¬ 
tions  are  concerned  with  the  technical  correctness  of  the  decisionmak¬ 
ing  procedures  embodied  within  the  tool  or  the  subjective  impressions 
of  the  suitability  of  the  human  user/decision  tool  interface.  Those  few 
evaluations  concerned  with  the  value  of  a  decision  aid  have  usually 
been  conducted  in  constrained  environments,  concentrating  on  the  per¬ 
formance  improvements  of  the  specific  decisionmaker.  Although  we 
will  also  measure  local  effects  during  the  postdevelopment  evaluations, 
we  believe  that  the  complex  nature  of  the  WTADS  requires  an  evalua¬ 
tion  of  the  global  system  performance. 

The  approach  we  will  adopt  to  conduct  postdevelopment  evaluations 
is  to  replace  one  of  the  simulated  objects  with  a  human  decisionmaker. 
We  can  then  use  the  WTADS  model  as  a  driver  to  present  to  the 
human  decisionmaker  information  on  the  status  of  the  system  and  the 
results  of  the  decisions  he  makes.  We  will  run  these  experiments  with 
the  decisionmaker  using  current  practices  and  measure  local  and 
overall  system  performance.  We  will  then  replicate  the  experiment  but 
provide  the  decisionmaker  with  the  decision  tool  to  assist  in  his 
decisionmaking  processes.  Comparing  the  performance  of  the  unaided 
and  aided  experiments  will  allow  us  to  measure  the  increase  in  perfor¬ 
mance  resulting  from  the  use  of  the  decision  aid. 

There  are  two  technical  challenges  in  this  approach  to  postdevelop¬ 
ment  evaluation.  We  must  first  ensure  that  the  experimental  design 
eliminates  or  reduces  biases  and  extraneous  effects  so  that  we  have 
confidence  that  the  system  performance  measures  result  purely  from 
the  presence  or  absence  of  the  decision  aid.  For  example,  the  data  and 
information  displayed  to  the  human  decisionmaker  during  the  experi¬ 
ments  must  replicate  the  data  and  information  that  would  actually  be 
available  during  wartime.  Also,  we  must  ensure  that  the  decisionmaker 
is  adequately  trained  in  using  the  WTADS  model  and  the  decision  aid 
to  reduce  the  effects  of  any  “learning”  the  decisionmaker  achieves  dur¬ 
ing  the  experiments.  These  are  just  some  examples  of  the  issues  we 
must  address  in  designing  the  experiments  for  postdevelopment  evalua¬ 
tions. 

A  second  technical  challenge  is  the  procedure  for  interfacing  the 
human  decisionmaker  with  the  WTADS  model.  This  issue  was  also 
raised  in  Sec.  Ill  when  discussing  the  cultivation  of  new  expertise. 
Because  humans  must  interact  with  the  simulation  model,  we  are 
designing  a  laboratory  environment  that  will  permit  the  types  of 
research  we  envision.  The  concepts  and  operations  of  this  Decision 
Support  System  (DSS)  analysis  laboratory  will  be  described  later  in 
this  section. 
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IMPLEMENTATION  OF  DECISION  AIDS 

The  final  research  question  is  how  to  implement  decision  aids  after 
they  have  been  developed,  or  more  specifically,  how  improved  manage¬ 
ment  decisionmaking  tools  should  be  introduced  to  maximize  their 
effectiveness  and  use.  A  common  problem  mentioned  in  the  literature 
on  implementing  decision  support  systems  is  the  overall  lack  of  user 
acceptance,'  resulting  in  expensive  decision  support  tools,  sitting 
unused  on  managers’  shelves. 

The  goal  of  this  aspect  of  the  research  is  to  determine  the  best  strat¬ 
egy  for  implementing  a  decision  aid  before  it  is  actually  fielded.  Of 
particular  concern  is  the  interface  between  the  human  decisionmaker 
and  the  decision  aid  in  terms  of  input  requirements,  output  displays, 
and  system  prompts  or  help  functions.  Specific  questions  center 
around  what  data  should  be  displayed,  how  the  data  should  be 
displayed,  and  what  data  manipulation  functions  should  be  provided  to 
the  user. 

Our  approach  in  examining  implementation  strategies  is  similar  to 
the  approach  we  will  use  for  postdevelopment  evaluations.  We  will 
interface  a  human  decisionmaker  with  the  WTADS  simulation  model 
and  observe  and  measure  user  performance  for  various  methods  of 
implementation.  We  will  follow  these  controlled  experiments  with 
predesigned  questionnaires  to  record  the  subjective  user  evaluations  of 
the  implementation  method.  As  with  our  other  research  objectives  and 
approaches,  there  are  a  number  of  technical  challenges. 

One  primary  concern  is  identifying  a  representative  set  of  implemen¬ 
tation  options.  We  will  develop  an  initial  list  of  different  ways  to 
implement  a  decision  support  tool,  but  we  expect  the  list  to  grow  as  we 
receive  user  impressions  and  feedback.  A  second  concern  is  the  evalua¬ 
tion  of  the  subjective  inputs  from  the  users  on  individual  implementa¬ 
tion  methods.  The  questionnaire  will  include  a  range  of  categories 
addressing  different  aspects  of  the  user/decision  tool  interface.  The 
categories  and  the  subjective  user  responses  suggest  that  a  multi¬ 
attribute  utility  assessment  technique  will  be  required.  The  final  tech¬ 
nical  question,  raised  previously  during  our  descriptions  of  the  post- 
development  evaluations,  is  the  procedure  for  interfacing  the  human 
decisionmaker  with  the  WTADS  simulation  model.  The  concepts  and 
operations  of  a  laboratory  environment  to  conduct  these  human/model 
interfaces  and  experiments  is  described  next. 


'The  lack  of  integration  of  new  decision  aids  into  the  current  system  and  the  lack  of 
continued  funding  and  maintenance  of  the  decision  support  tools  are  also  reasons  that 
new  systems  go  unused. 
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THE  OBJECTIVES  OF  A  USS  ANALYSIS  LABORATORY 

A  number  of  our  research  objectives,  as  described  in  this  section  and 
the  previous  one,  require  the  interaction  of  a  human  decisionmaker 
with  the  WTADS  simulation  model.  Specifically,  we  need  to  interface 
a  human  with  the  model  in  order  to; 

•  Extend  existing  WTADS  management  expertise  by  developing 
heuristic  decision  rules 

•  Perform  postdevelopment  evaluations  of  specific  decision  sup¬ 
port  tools 

•  Examine  and  evaluate  different  methods  for  implementing  new 
decision  support  tools. 

The  structure  of  the  ROSS  language,  and  the  resulting  WTADS 
simulation  model,  provide  a  ready  and  effective  mechanism  for  human 
interfaces.  Unlike  conventional  simulation  languages,  such  as  FOR¬ 
TRAN  or  SIMSCRIPT,  the  physical  “objects”  in  an  object-oriented 
simulation  model  are  really  small  “simulations,”  containing  all  the  data 
and  logic  peculiar  to  that  “object.”  Therefore,  any  of  the  objects  or 
actors  programmed  in  the  model  can  be  replaced  by  interactive  inputs 
in  terms  of  responses  to  messages  directed  to  that  object.  For  example, 
Wl’ADS  contains  the  attributes  and  behaviors  that  reflect  the  manage¬ 
ment  decisionmaking  procedures  of  the  Division  Ammunition  Officer 
(DAO).  As  the  simulation  is  running,  the  DAO  object  receives  various 
messages  and  information  from  other  objects  in  the  model.  Based  on 
the  set  of  responses  contained  in  the  model,  the  DAO  object  generates 
its  own  messages  reflecting  decisions  that  it  must  make.  With  ROSS, 
this  DAO  object  can  be  represented  by  inputs  into  the  model.  There¬ 
fore,  a  human  can  be  seated  at  a  terminal  receiving  model  output  and 
provide  corresponding  input  representing  the  DAO’s  behavior. 

By  interfacing  a  human  with  the  simulation  model,  we  can  represent 
an  environment  in  which  an  ammunition  distribution  manager  can  see 
the  combat  effects  of  his  decisions.  By  replicating  experiments,  the 
decisionmaker  can  examine  the  benefits  or  limitations  of  various  deci¬ 
sion  strategies,  eventually  developing  expertise  in  his  domain.  The  end 
result  should  be  an  increased  set  of  heuristic  procedures  for  dealing 
with  the  complexities  and  uncertainties  that  will  arise  in  wartime. 

Such  a  laboratory  environment  would  also  form  the  basis  for  a 
mechanism  to  identify  and  evaluate  general  types  of  information  and 
decisionmaking  improvements  in  the  ammunition  distribution  area. 
This  capability  would  include  the  a  priori  evaluation  of  proposed  deci¬ 
sion  support  tools,  the  examination  of  the  effects  of  improved  informa¬ 
tion  content  or  reduced  information  “noise,”  and  the  evaluation  of 
better  short  time  horizon  forecasting  capabilities. 
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The  initial  hardware  concepts  for  the  laboratory  include  a  number  of 
terminals  connected  to  a  host  computer  that  will  execute  the  WTADS 
model.  A  representation  of  the  laboratory  structure  is  shown  in  Fig.  7. 
A  human  controller  and  knowledge  engineers,  in  concert  with  a 
scenario  generator,  will  interact  with  the  WTADS  model  to  present 
combat  situations  to  a  human  decisionmaker.  The  terminals  and  other 
display  devices  will  graphically  represent  various  logistics  data  and  bat¬ 
tlefield  information.  One  technical  concern  is  how  these  data  can  best 
be  represented  in  order  to  provide  the  necessary  information  and  still 
realistically  portray  the  data  flows  that  will  occur  during  wartime.  It 
may  be  necessary  to  deliberately  corrupt  the  data  to  attain  the  proper 
degree  of  wartime  “realism.”  The  human  decisionmaker,  based  on  his 
perceptions  of  the  course  of  the  battle  and  the  data  available,  will  com¬ 
municate  his  decisions  to  the  WTADS  model  via  a  terminal. 

To  capture  the  specific  nuances  of  the  decisionmaking  process  dur¬ 
ing  laboratory  experiments,  the  environment  will  include  various 
audio-visual  recording  and  playback  equipment.  This  aspect  of  the 
laboratory  is  especially  critical  for  capturing  the  protocols  of  the 
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Fig.  7 — Prototype  DSS  laboratory  structure 
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decisionmakers  as  they  solve  various  problems.  Protocol  analysis  is  a 
necessary  step  in  the  development  of  any  decision  support  system. 

Our  initial  objective  in  developing  the  laboratory  is  to  examine  our 
immediate  research  problems.  Although  we  will  begin  the  development 
of  the  laboratory  on  a  narrow  basis,  we  believe  that  such  an  environ¬ 
ment  can  easily  be  extended  to  encompass  additional  objectives.  For 
example,  the  laboratory  can  provide  the  Army  with  a  prototype  meth¬ 
odology  for  developing  decision  support  enhancements  at  relatively  low 
cost,  in  addition  to  being  adaptable  as  an  Army  ammunition  manager 
training  device.  The  laboratory  could  also  provide  a  training  environ¬ 
ment  for  various  WTADS  managers.  Finally,  the  laboratory  could  be 
extended  to  include  other  functional  areas,  such  as  supply  and  repair. 
Such  extensions  could  ultimately  lead  to  the  foundations  of  a  logistic 
wargaming  capability. 
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